K-State Honor Code:
On my honor, as a student, I have neither given nor received unauthorized aid on this academic work." A grade of XF can result from a breach of academic honesty
Your name: Mohammad Najjartabar Bisheh
In [1]:
from IPython.display import Image
from IPython.core.display import HTML
Image(url= "https://i.pinimg.com/originals/db/4f/88/db4f88f155d22599f59765e14f4c5497.jpg")
Out[1]:
1. Business understanding:¶
Movies are an important part of human's life these days. We can name movies as the cheapest and most available fun that everyone can have and it is not depended to how wealthy we are. A great movie can entertain almost everyon on this planet. But what makes a movie great? A good story? Great characters? cinematography and actors' performance? The answer is yes, all of these can impact on the quality of a movie. The important question is how we can measure quality and perfomrance of a movie?¶
Quality and performance of a movie can be measured in two point of view. First of all and as a general idea, critics review's can be a good refference to judge about quality of a movie. Critics will consider quality of a movie by their own criteria (e.g. Directing, Writing, Cinematography, Editing, Acting, Production Design, Sound and etc.). However to talk about performance of a movie, "perceived performance of audience" plays an important rule. To measure "perceived performance of audience" maybe we can simply look at amount of money that movie made but what have impact on the sell of a movie?¶
Todays, there are some websites (like IMDB) which does movie rating usually base on peoples' weighted vote. In this project, (based on some given information like genre, budget, duration, and ...) I want to see what is impact of each (and maybe combination of some) factor on IMDB rating which can assume as "perceived performance of audience".¶
In [2]:
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
# import the scatter_matrix functionality
from pandas.plotting import scatter_matrix
import plotly.graph_objects as go
import plotly.express as px
import pingouin as pg
import statsmodels.api as sm
from statsmodels.formula.api import ols
#regression packages
import sklearn.linear_model as lm
from sklearn.metrics import mean_squared_error
from sklearn.metrics import explained_variance_score
#lasso regression
from sklearn import linear_model
#f_regression (feature selection)
from sklearn.feature_selection import f_regression
from sklearn.feature_selection import SelectKBest
# recursive feature selection (feature selection)
from sklearn.feature_selection import RFE
from sklearn.cluster import KMeans
from sklearn.cluster import AgglomerativeClustering
from sklearn.metrics import pairwise_distances
# Classifiers
#import decisiontreeclassifier
from sklearn import tree
from sklearn.tree import DecisionTreeClassifier
#import logisticregression classifier
from sklearn.linear_model import LogisticRegression
import statsmodels.api as sm
#import knn classifier
from sklearn.neighbors import KNeighborsClassifier
#for validating your classification model
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn import metrics
from sklearn.metrics import roc_curve, auc
from sklearn.metrics import roc_auc_score
# feature selection
from sklearn.feature_selection import RFE
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
# grid search
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
2. Data understanding: The dataset contains a large number of variables with different types (e.g., numerical, categorial). Provide a brief summary of data understanding. Specifically, you need to:¶
In [3]:
# Reading dataset
df = pd.read_csv('movie_metadata.csv\movie_metadata.csv')
df.head()
Out[3]:
Just with a look at the head of data we can see there are NaN and zero values which means missing data. df.info proves this that we are missing many values and data needs to be cleaned.¶
In [4]:
df.info()
In [5]:
# Describe data
df.describe()
Out[5]:
data quality issues¶
(William McKnight, in Information Management, 2014)
Sources of Poor Data Quality The following are seven sources of data quality issues.
1.Entry quality: Did the information enter the system correctly at the origin?
2.Process quality: Proper checks and quality control at each touchpoint along the path can help ensure that problems are rooted out, but these checks are often absent in legacy processes.
3.Identification quality: Data quality processes can largely eliminate this problem by matching records, identifying duplicates, and placing a confidence score4 on the similarity of records.
4.Integration quality: Is all the known information about an object integrated to the point of providing an accurate representation of the object?
5.Usage quality: Is the information used and interpreted correctly at the point of access?
6.Aging quality: Has enough time passed that the validity of the information can no longer be trusted?
7.Organizational quality: The biggest challenge to reconciliation is getting the various departments to agree that their A equals the other’s B equals the other’s C plus D.
Based on our current dataset, we need to take care of missing and duplicated values.¶
In [6]:
#Finding and deleting duplicated values
len(df[df.duplicated()])
Out[6]:
In [7]:
len(df)
Out[7]:
In [8]:
# Finding and taking care of missing values
df.isnull().info()
See if there is any zeros in the table¶
In [9]:
criteria1 = df['budget'] == 0;
criteria2 = df['gross'] == 0;
criteria3 = df['color'] == 0;
criteria4 = df['director_name'] == 0;
criteria5 = df['num_critic_for_reviews'] == 0;
criteria6 = df['duration'] == 0;
criteria7 = df['director_facebook_likes'] == 0;
criteria8 = df['actor_3_facebook_likes'] == 0;
criteria9 = df['actor_2_name'] == 0;
criteria10 = df['actor_1_facebook_likes'] == 0;
criteria11 = df['genres'] == 0;
criteria12 = df['actor_1_name'] == 0;
criteria13 = df['genres'] == 0;
criteria14 = df['movie_title'] == 0;
criteria15 = df['num_voted_users'] == 0;
criteria16 = df['cast_total_facebook_likes'] == 0;
criteria17 = df['actor_3_name'] == 0;
criteria18 = df['facenumber_in_poster'] == 0;
criteria19 = df['plot_keywords'] == 0;
criteria20 = df['movie_imdb_link'] == 0;
criteria21 = df['num_user_for_reviews'] == 0;
criteria22 = df['language'] == 0;
criteria23 = df['content_rating'] == 0;
criteria24 = df['title_year'] == 0;
criteria25 = df['actor_2_facebook_likes'] == 0;
criteria26 = df['imdb_score'] == 0;
criteria26 = df['movie_facebook_likes'] == 0;
criteria = criteria1 & criteria2 & criteria3 & criteria4 & criteria5 & criteria6 & criteria7 & criteria8 & criteria9 & criteria10 & criteria11 & criteria12 & criteria13 & criteria14 & criteria15 & criteria16 & criteria17 & criteria18 & criteria19 & criteria20 & criteria21 & criteria22 & criteria23 & criteria24 & criteria25 & criteria26
In [10]:
# There are some missing values and marked by “”. I replace them with "na"
df['content_rating'] = df['content_rating'].str.replace('“”', 'na')
df['content_rating'] = df['content_rating'].str.replace('0', 'na')
In [11]:
missing = df.isna()
missing2 = missing.sum()
missing2 = missing2.reset_index(level=0)
missing2 = missing2.rename(columns={0: 'factor', 1: 'total'})
missing2 = missing2.set_index('index')
missing2 = missing2.sort_values('factor', ascending=False).plot(kind='bar')
# if I want to do analysis which contain gross, budget, aspect_ratio or content rating I will use dff which is df without "NA".
More than 800 movies we don't have their gross income and about 500 movies we don't have their budget. Also cosiderable missing values in aspect ratio and content rating and all of these factors are important in IMDB score analysis so we need to do something.¶
In [12]:
missing['actor_1_name'].describe()
Out[12]:
In [13]:
df['gross'].describe()
Out[13]:
In [14]:
df['director_name'].value_counts().head(10)
Out[14]:
In [15]:
df['actor_1_name'].value_counts().head(10)
Out[15]:
In [16]:
df['budget'].value_counts().head()
Out[16]:
These are some general information about some selected columns to get a rough idea about them!¶
In [17]:
df = df.drop_duplicates(keep='first')
len(df)
Out[17]:
In [18]:
df.dropna(how='all');
df.dropna(subset=['imdb_score'])
df.info()
Our dataset in df is not constant but I don't want to lose it because we can have good analysis on it.¶
In [19]:
# To have a constant dataset I made dff:
dff = df.dropna()
dff.info()
In [20]:
b = pd.DataFrame(dff.genres.str.split('|').tolist(), index=dff.imdb_score).stack()
b = b.reset_index()[[0, 'imdb_score']] # genres variable is currently labeled 0
b.columns = ['genres', 'imdb_score'] # renaming genres
b.head()
Out[20]:
b is a dataframe which contains movies based on their unique genres¶
In [21]:
# just a different way
df2 = dff.join(dff.pop('genres').str.get_dummies('|'))
df2.head()
Out[21]:
In [22]:
b['genres'].value_counts().plot(kind='bar')
Out[22]:
In [23]:
b.groupby('genres')['imdb_score'].mean().plot.bar()
Out[23]:
These plots shows most of the genres are drama, comedy, thriller or action but they don't usually have the highest IMDB value¶
In the example they digged in to content_rating and tried to replace some values with other values. I really don't see any necessity to do that but to show that I could do that I'll do and make a different dataframe for it!¶
In [24]:
dff['content_rating'].value_counts()
Out[24]:
In [25]:
#First method
dff_replace = dff.replace({'content_rating': 'GP'}, {'content_rating': 'R'});
dff_replace = dff_replace.replace({'content_rating': 'M'}, {'content_rating': 'R'});
dff_replace = dff_replace.replace({'content_rating': 'Passed'}, {'content_rating': 'R'});
dff_replace = dff_replace.replace({'content_rating': 'NC-17'}, {'content_rating': 'R'});
dff_replace['content_rating'].value_counts()
Out[25]:
In [26]:
#second method
# create a function
def f(x):
if x == 'R': return 'R'
elif x == 'PG-13': return 'PG-13'
elif x == 'PG': return 'PG'
elif x == 'G': return 'G'
elif x == 'Not Rated': return 'Not Rated'
elif x == 'Unrated': return 'Unrated'
elif x == 'Approved': return 'Approved'
elif x == 'X': return 'X'
else: return 'R'
In [27]:
dff_replace['content_rating'] = dff['content_rating'].apply(f)
dff_replace['content_rating'].value_counts()
Out[27]:
So we have a clean data now and we can go ahead and do some analysis!¶
Business intelligence¶
first of all to compare success in terms of money we need to know about profit of each movie. Since we have gross (income) and budget then we can find profit which is equal to gross - profit¶
In [28]:
dff['profit'] = dff ['gross'] - dff['budget']
dff.info()
In [29]:
dff['return_on_investment_perc'] = (dff ['profit'] / dff['budget']) * 100
dff.info()
In [30]:
#dff = dff.set_index('movie_title');
dff['profit'].head()
Out[30]:
In [31]:
best_profit = dff.pivot_table(index='movie_title', aggfunc='sum', fill_value=0).sort_values(by=['profit'], ascending=False).head(20)
best_profit.head()
Out[31]:
In [32]:
plt.figure(figsize=[16,16])
pieprofit = dff[['movie_title','profit','budget','return_on_investment_perc', 'gross','imdb_score']];
pieprofit = pieprofit.sort_values(by=['profit'], ascending = False).head(10);
explode = (0.2, 0, 0, 0, 0.1, 0, 0, 0.15, 0, 0) # explode 1st slice
colors = ['gold', 'yellowgreen', 'lightcoral', 'lightskyblue','maroon', 'aqua', 'khaki', 'darkturquoise', 'hotpink', 'mediumpurple']
plt.pie(pieprofit['profit'], labels=pieprofit['movie_title'], explode=explode, colors=colors, autopct='%1.1f%%', shadow=True, startangle=140)
plt.axis('equal')
Out[32]:
So Avatar, Jurassic World, Titanic and Starwas: Episode IV had the best profit.¶
In [33]:
dff['color'].value_counts().head()
Out[33]:
almost all of movies are color and just 123 of them are black and white. we can remove color column here but personally I don't like miss any information so I'll keep it.¶
In [34]:
plt.figure(figsize=[16,16])
piereturn = dff[['movie_title','profit','budget','return_on_investment_perc', 'gross','imdb_score']];
piereturn = piereturn.sort_values(by=['return_on_investment_perc'], ascending = False).head(10);
explode = (0.2, 0, 0, 0, 0.1, 0, 0, 0.15, 0, 0) # explode 1st slice
colors = ['gold', 'yellowgreen', 'lightcoral', 'lightskyblue','maroon', 'aqua', 'khaki', 'darkturquoise', 'hotpink', 'mediumpurple']
plt.pie(piereturn['return_on_investment_perc'], labels=piereturn['movie_title'], explode=explode, colors=colors, autopct='%1.1f%%', shadow=True, startangle=140)
plt.axis('equal')
Out[34]:
In [35]:
piereturn = dff[['movie_title','profit','budget','return_on_investment_perc', 'gross','imdb_score']];
piereturn = piereturn.sort_values(by=['return_on_investment_perc'], ascending = False).head(20)
#px.scatter(piereturn, x="budget", y="return_on_investment_perc", text ='movie_title' , trendline="lowess")
piereturn
Out[35]:
Is language an important factor for imdb score? What about country?¶
In [36]:
#let's get some general information from language and country
dff['language'].value_counts().head(16)
Out[36]:
In [37]:
dff['country'].value_counts().head(20)
Out[37]:
In [38]:
e = dff.loc[df['country'] == 'Iran']
e.groupby(['movie_title','imdb_score'])['profit'].sum().sort_values(ascending=False).head()
Out[38]:
In [39]:
e = dff.loc[df['language'] == 'Persian']
e.groupby(['movie_title','imdb_score'])['profit'].sum().sort_values(ascending=False).head()
Out[39]:
Ok now I found the problem. Caravans was shot in Afghanistan and iran and starred Anthony Quinn, Jennifer O'Neill, and Michael Sarrazin.¶
In [40]:
# I want to draw pareto chart
def pareto_plot(df, x=None, y=None, title=None, show_pct_y=False, pct_format='{0:.0%}'):
xlabel = x
ylabel = y
tmp = df.sort_values(y, ascending=False)
x = tmp[x].values
y = tmp[y].values
weights = y / y.sum()
cumsum = weights.cumsum()
fig, ax1 = plt.subplots()
ax1.bar(x, y)
ax1.set_xlabel(xlabel)
ax1.set_ylabel(ylabel)
ax2 = ax1.twinx()
ax2.plot(x, cumsum, '-ro', alpha=0.5)
ax2.set_ylabel('', color='r')
ax2.tick_params('y', colors='r')
vals = ax2.get_yticks()
ax2.set_yticklabels(['{:,.2%}'.format(x) for x in vals])
# hide y-labels on right side
if not show_pct_y:
ax2.set_yticks([])
formatted_weights = [pct_format.format(x) for x in cumsum]
for i, txt in enumerate(formatted_weights):
ax2.annotate(txt, (x[i], cumsum[i]), fontweight='heavy')
if title:
plt.title(title)
plt.tight_layout()
plt.show()
In [41]:
pareto = dff.sort_values(by=['budget'], ascending = False).head(8)
In [42]:
pareto_plot(pareto, x='movie_title', y='gross', title='profit pareto')
In [43]:
dff.groupby(['movie_title','imdb_score'])['profit'].sum().sort_values(ascending=False).head(10)
Out[43]:
In [44]:
dff.groupby(['movie_title','profit'])['imdb_score'].sum().sort_values(ascending=False).head(10)
Out[44]:
woww. "Eddie the Eagle" with negative profit had a great score! That doesn't make sense! Maybe their marketin was not good!¶
In [45]:
fig = px.scatter(dff, x="imdb_score", y="gross")
fig.add_trace(
go.Scatter(
x=[8, 8],
y=[0, 800000000],
mode="lines",
line=go.scatter.Line(color="gray"),
showlegend=False)
)
fig.add_trace(
go.Scatter(
x=[1, 10],
y=[500000000, 500000000],
mode="lines",
line=go.scatter.Line(color="gray"),
showlegend=False)
)
fig.show()
In [46]:
fig = px.scatter(pieprofit, x="imdb_score", y="gross", text ='movie_title')
fig.add_trace(
go.Scatter(
x=[8, 8],
y=[50000000, 800000000],
mode="lines",
line=go.scatter.Line(color="gray"),
showlegend=False)
)
fig.add_trace(
go.Scatter(
x=[6, 9],
y=[500000000, 500000000],
mode="lines",
line=go.scatter.Line(color="gray"),
showlegend=False)
)
fig.show()
So Avatar had the best profit but The Dark Night had the best IMDB score!¶
In [47]:
df1 = dff
# setting my own values for bins
df1['imdbscores_bins'] = pd.cut(df1['imdb_score'], bins=[0, 2, 4, 6, 8, 10], labels=['0-1.99', '2-3.99', '4-5.99', '6-7.99', '8-10'],
include_lowest=True)
# see the result
df1.head()
Out[47]:
Here I made bins for IMDB scores and changed their name.¶
In [48]:
#labels=['trash', 'tolerable', 'good', 'accaptable', 'incredible', 'unbelievable']
df1 = df1.replace({'imdbscores_bins': "6-7.99"}, {'imdbscores_bins': 'incredible'})
df1 = df1.replace({'imdbscores_bins': "0-1.99"}, {'imdbscores_bins': 'trash'})
df1 = df1.replace({'imdbscores_bins': "2-3.99"}, {'imdbscores_bins': 'tolerable'})
df1 = df1.replace({'imdbscores_bins': "4-5.99"}, {'imdbscores_bins': 'accaptable'})
df1 = df1.replace({'imdbscores_bins': "8-10"}, {'imdbscores_bins': 'unbelievable'})
df1.head()
Out[48]:
In [49]:
fig = px.scatter_3d(dff, x='return_on_investment_perc', y='country', z='imdb_score')
fig.show()
In [50]:
fig = px.scatter_3d(df1, x='return_on_investment_perc', y='country', z='imdbscores_bins')
fig.show()
a huge number of movies has imdb better than 8 which I named as incredible¶
In [51]:
fig = px.scatter_3d(pieprofit, x='profit', y='return_on_investment_perc', z='imdb_score', text = 'movie_title')
fig.show()
In [52]:
fig = px.scatter(dff, x="country", y="imdb_score", marginal_y="rug", marginal_x="histogram")
fig.show()
In [53]:
fig = px.scatter(dff, x="gross", y="imdb_score", marginal_y="violin",
marginal_x="box", trendline="ols")
fig.show()
In [54]:
plt.figure(figsize=[24,16])
fig = px.scatter_matrix(dff, dimensions=["imdb_score", "actor_1_facebook_likes", "actor_2_facebook_likes"])
fig.show()
I don't see any strong relations between actors facebook likes and imdb score¶
In [55]:
df1['imdbscores_bins'].value_counts().head()
Out[55]:
In [57]:
fig = px.parallel_coordinates(dff, color="imdb_score", labels={"actor_1_facebook_likes": "Actor 1 likes",
"actor_2_facebook_likes": "Actor 2 likes", "actor_3_facebook_likes": "Actor 3 likes",
"director_facebook_likes": "Director likes", },
color_continuous_scale=px.colors.diverging.Tealrose, color_continuous_midpoint=2)
fig.show()
In [58]:
earth = dff.groupby(['country', 'gross'])['title_year'].sum(
).sort_values()
earth.head()
Out[58]:
In [59]:
earth2=dff.pivot_table(index=['country','title_year'], values='gross',
aggfunc='sum', fill_value=0, margins=True).reset_index()
#earth2=earth2['country'].value_counts()
earth2.head()
Out[59]:
In [60]:
fig = px.choropleth(earth2, locations="country", color="gross", hover_name="country", animation_frame="title_year", range_color=[0,999999999])
fig.show()
I gave a shot but didn't work. Didn't have enough time otherwise I could make it¶
In [61]:
#dff.groupby('imdb_score').hist(figsize=(10,10));
df1.groupby('imdbscores_bins')['actor_1_facebook_likes'].sum().plot.bar();
In [62]:
df1.groupby('imdbscores_bins')['actor_2_facebook_likes'].sum().plot.bar();
In [63]:
df1.groupby('imdbscores_bins')['actor_3_facebook_likes'].sum().plot.bar();
In [64]:
sns.lmplot("actor_1_facebook_likes", "director_facebook_likes", df1, hue="imdbscores_bins", x_jitter=.15, height=8)
Out[64]:
Now check each variable with imdb score¶
In [65]:
#ff.groupby('imdbscores_bins')['duration'].sum().plot.bar();
sns.violinplot("duration", "imdbscores_bins", data=df1,
palette=["lightblue", "lightpink"])
Out[65]:
In [66]:
#dff.groupby('imdbscores_bins')['num_critic_for_reviews'].sum().plot.bar();
sns.violinplot("num_critic_for_reviews", "imdbscores_bins", data=df1,
palette=["lightblue", "lightpink"])
Out[66]:
Correlation analysis¶
In [67]:
dffcorr = dff.corr()
dffcorr
Out[67]:
In [68]:
dffcorr ['imdb_score']
Out[68]:
So briefly we can see here that num_critic_for_reviews, duration, num_voted_users, num_user_for_reviews, and movie_facebook_likes are highly positive correlated to imdb score and title year is negativly and relativly correlated which means people probably won't like the most recent movies as much as they liked old movies but probably fact is each movies takes times to get more viewers and increase imdb score and older movies had enough time for that¶
In [69]:
plt.figure(figsize=(14,10))
sns.heatmap(dff.corr(), vmax=.8, square=True, annot=True, fmt=".1f")
Out[69]:
In [70]:
sns.jointplot("imdb_score", "duration", dff, kind="hex", color="#8855AA")
Out[70]:
In [71]:
sns.jointplot("imdb_score", "num_critic_for_reviews", dff, kind="hex", color="#8855AA")
Out[71]:
In [72]:
sns.jointplot("imdb_score", "num_user_for_reviews", dff, kind="hex", color="#8855AA")
Out[72]:
ANOVA test¶
In [73]:
cw_lm=ols('imdb_score ~ num_critic_for_reviews + duration + director_facebook_likes + actor_3_facebook_likes + actor_1_facebook_likes + gross + num_voted_users + cast_total_facebook_likes + facenumber_in_poster + num_user_for_reviews + budget + actor_2_facebook_likes + actor_2_facebook_likes + aspect_ratio + movie_facebook_likes + profit + return_on_investment_perc',
data=dff).fit() #Specify C for Categorical
print(sm.stats.anova_lm(cw_lm, typ=2))
In [75]:
# Anova test
cw_lm2=ols('imdb_score ~ num_critic_for_reviews + duration + num_voted_users + num_user_for_reviews + movie_facebook_likes',
data=dff).fit() #Specify C for Categorical
print(sm.stats.anova_lm(cw_lm2, typ=2))
In [76]:
runs_reg_model1 = ols("imdb_score ~ num_critic_for_reviews + duration + num_voted_users + num_user_for_reviews + movie_facebook_likes",dff)
runs_reg1 = runs_reg_model1.fit()
print(runs_reg1.summary())
In [77]:
runs_reg1.mse_resid
Out[77]:
In [78]:
runs_reg_model = ols("imdb_score ~ profit",dff)
runs_reg2 = runs_reg_model.fit()
print(runs_reg1.summary())
In [79]:
runs_reg2.mse_resid
Out[79]:
In [80]:
runs_reg_model = ols("imdb_score ~ num_critic_for_reviews",dff)
runs_reg3 = runs_reg_model.fit()
print(runs_reg1.summary())
In [81]:
runs_reg3.mse_resid
Out[81]:
In [82]:
runs_reg_model = ols("imdb_score ~ duration",dff)
runs_reg4 = runs_reg_model.fit()
print(runs_reg1.summary())
In [83]:
runs_reg4.mse_resid
Out[83]:
In [84]:
runs_reg_model = ols("imdb_score ~ num_voted_users",dff)
runs_reg5 = runs_reg_model.fit()
print(runs_reg1.summary())
In [85]:
runs_reg5.mse_resid
Out[85]:
In [86]:
runs_reg_model = ols("imdb_score ~ num_user_for_reviews",dff)
runs_reg6 = runs_reg_model.fit()
print(runs_reg1.summary())
In [87]:
runs_reg6.mse_resid
Out[87]:
In [88]:
runs_reg_model = ols("imdb_score ~ movie_facebook_likes",dff)
runs_reg7 = runs_reg_model.fit()
print(runs_reg1.summary())
In [89]:
runs_reg7.mse_resid
Out[89]:
all the above regression models has same r-squared valed (.304) but the full model has the lowest mean squared value which means full model is better. Also all p-values are small so all the selected fatcors are significant.¶
Regression model based on regularization¶
In [90]:
df5 = dff[['imdb_score', 'num_critic_for_reviews','duration','num_voted_users','num_user_for_reviews', 'movie_facebook_likes']];
In [91]:
y = df5['imdb_score']
X = df5.drop(['imdb_score'], axis =1)
model1 = linear_model.Lasso(alpha=1)
model1.fit(X, y)
model1_y = model1.predict(X)
print('Coefficients: ', model1.coef_)
print("y-intercept ", model1.intercept_)
In [92]:
coef = ["%.3f" % i for i in model1.coef_]
xcolumns = [ i for i in X.columns ]
list(zip(xcolumns, coef))
Out[92]:
In [93]:
print("mean square error: ", mean_squared_error(y, model1_y))
print("variance or r-squared: ", explained_variance_score(y, model1_y))
This model gave same MSE but lower r-squered than privous one.¶
b) Regression model based on Feature selection¶
In [94]:
X_new = SelectKBest(f_regression, k=2).fit_transform(X, y)
X_new
Out[94]:
In [95]:
# this helps us find out which variables are selected
selector = SelectKBest(f_regression, k=2).fit(X, y)
idxs_selected = selector.get_support(indices=True)
print(idxs_selected)
In [96]:
model2 = lm.LinearRegression()
model2.fit(X_new, y)
model2_y = model2.predict(X_new)
print("mean square error: ", mean_squared_error(y, model2_y))
print("variance or r-squared: ", explained_variance_score(y, model2_y))
In [97]:
selector = SelectKBest(f_regression, k=3).fit(X, y)
idxs_selected = selector.get_support(indices=True)
model3 = lm.LinearRegression()
model3.fit(X_new, y)
model3_y = model3.predict(X_new)
print("mean square error: ", mean_squared_error(y, model3_y))
print("variance or r-squared: ", explained_variance_score(y, model3_y))
Here neither the MSE nor r-squared are better than first method!¶
Hierarchical Dendrogram¶
In [98]:
from sklearn.cluster import AgglomerativeClustering
from sklearn.cluster import ward_tree
from scipy.cluster.hierarchy import dendrogram, linkage, ward
from sklearn.metrics import pairwise_distances
In [99]:
np.random.seed(1) # setting random seed to get the same results each time.
agg= AgglomerativeClustering(n_clusters=4, linkage='ward').fit(X)
agg.labels_
Out[99]:
In [100]:
plt.figure(figsize=(16,8))
linkage_matrix = ward(X)
dendrogram(linkage_matrix, orientation="top")
plt.tight_layout() # fixes margins
In [101]:
plt.figure(figsize=(16,8))
plt.title('Hierarchical Clustering Dendrogram (truncated)')
plt.xlabel('sample index or (cluster size)')
plt.ylabel('distance')
linkage_matrix = ward(X)
dendrogram(linkage_matrix,
truncate_mode='lastp', # show only the last p merged clusters
p=4, # show only the last p merged clusters
#show_leaf_counts=False, # otherwise numbers in brackets are counts
leaf_rotation=90.,
leaf_font_size=12.,
show_contracted=True, # to get a distribution impression in truncated branches
orientation="top")
plt.tight_layout() # fixes margins
PCA¶
In [102]:
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
pca.fit(X)
print(pca.explained_variance_ratio_)
print(pca.singular_values_)
In [103]:
pca = PCA(n_components=2, svd_solver='full')
pca.fit(X)
print(pca.explained_variance_ratio_)
print(pca.singular_values_)
In [104]:
pca = PCA(n_components=1, svd_solver='arpack')
pca.fit(X)
print(pca.explained_variance_ratio_)
print(pca.singular_values_)
In [105]:
from sklearn.decomposition import IncrementalPCA
ipca = IncrementalPCA(n_components=2, batch_size=3)
ipca.fit(X)
ipca.transform(X)
Out[105]:
Classification¶
To train classification model again I used the factors that was significant based on earlier analysis. Same as last section, I will run several machine learning model first and then talk about them at the end.¶
In [106]:
df6 = dff.dropna()
df6 = df6[['imdbscores_bins', 'num_critic_for_reviews','duration','num_voted_users','num_user_for_reviews', 'movie_facebook_likes']];
y = df6['imdbscores_bins']
X = df6.drop(['imdbscores_bins'], axis =1)
In [107]:
df6 = df.dropna()
df6 = df6[['imdb_score', 'num_critic_for_reviews','duration','num_voted_users','num_user_for_reviews', 'movie_facebook_likes']];
# setting my own values for bins
df6['imdbscores_bins'] = pd.cut(df6['imdb_score'], bins=[0, 4, 6, 8, 10], labels=[1, 2, 3, 4],
include_lowest=True)
# see the result
df6.info()
In [108]:
df6 = df6.drop(['imdb_score'], axis =1)
In [109]:
df6['imdbscores_bins'] = df6.imdbscores_bins.astype(int)
df6.info()
In [110]:
y = df6['imdbscores_bins']
X = df6.drop(['imdbscores_bins'], axis =1)
a) Decision Tree¶
In [111]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)
dt = DecisionTreeClassifier()
dt.fit(X_train, y_train);
In [112]:
print(metrics.accuracy_score(y_test, dt.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.confusion_matrix(y_test, dt.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.classification_report(y_test, dt.predict(X_test)))
print("--------------------------------------------------------")
#print(metrics.roc_auc_score(y_test, dt.predict(X_test)));
In [113]:
import scikitplot as skplt
skplt.metrics.plot_confusion_matrix(y_true=np.array(y_test), y_pred=dt.predict(X_test))
plt.show()
In [114]:
from sklearn.externals.six import StringIO
import pydotplus
dot_data = StringIO()
tree.export_graphviz(dt, out_file=dot_data, feature_names=X.columns,
filled=True, rounded=True, special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
In [115]:
from graphviz import Source
from sklearn import tree
Source( tree.export_graphviz(dt, out_file=None, feature_names=X.columns))
Out[115]:
b) KNN¶
In [116]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)
knn = KNeighborsClassifier()
knn.fit(X_train, y_train)
Out[116]:
In [117]:
print(metrics.accuracy_score(y_test, knn.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.confusion_matrix(y_test, knn.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.classification_report(y_test, knn.predict(X_test)))
print("--------------------------------------------------------")
#print(metrics.roc_auc_score(y_test, knn.predict(X_test)))
In [118]:
k_range = range(1, 10)
scores = []
for k in k_range:
knn = KNeighborsClassifier(n_neighbors=k)
scores.append(np.mean(cross_val_score(knn, X, y, cv=10, scoring = 'accuracy')))
#plt.figure()
plt.plot(k_range, scores)
plt.xlabel('k value')
plt.ylabel('accuracy')
Out[118]:
this graph shows the best number for k is 3 here.¶
c) GradientBoostingClassifier¶
In [119]:
from sklearn.neural_network import MLPClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC
In [120]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
gb = GradientBoostingClassifier(n_estimators=100, random_state=0)
gb.fit(X_train, y_train)
Out[120]:
In [121]:
print(metrics.accuracy_score(y_test, gb.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.confusion_matrix(y_test, gb.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.classification_report(y_test, gb.predict(X_test)))
print("--------------------------------------------------------")
In [122]:
# 10-fold cross-validation
scores = cross_val_score(gb, X, y, scoring='accuracy', cv=10)
print(scores)
print(scores.mean())
d) Support Vector Machine¶
In [123]:
svm = SVC(gamma='scale', probability=True)
svm.fit(X_train, y_train)
Out[123]:
In [124]:
print(metrics.accuracy_score(y_test, svm.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.confusion_matrix(y_test, svm.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.classification_report(y_test, svm.predict(X_test)))
print("--------------------------------------------------------")
In [125]:
# 10-fold cross-validation
svm = SVC(gamma='auto')
scores = cross_val_score(svm, X, y, scoring='accuracy', cv=10)
print(scores)
print(scores.mean())
e) Neural Network¶
In [126]:
nn = MLPClassifier(solver='lbfgs', max_iter=500)
nn.fit(X_train, y_train)
Out[126]:
In [127]:
print(metrics.accuracy_score(y_test, nn.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.confusion_matrix(y_test, nn.predict(X_test)))
print("--------------------------------------------------------")
print(metrics.classification_report(y_test, nn.predict(X_test)))
print("--------------------------------------------------------")
In this section I ran several classification models. Gradient boosting had the best performance with accruacy close to 70 percent and then decision tree had 62 percent accuracy. SVM gave 67 percent but actually it didn't work well based on confusion matrix. Both SVM and neural network had lazy performance which means they classified all test data set in one category (SVM classified as good and neural network classified as bad).¶
Clustering¶
In this section we want to implement unsupervised machine learning and I did Kmeans which is the most popular algorithm in this area. First I ran a Kmeans with four clusters and then I used Silhoutte and elbow method to find the best value for K.¶
In [128]:
dfff = df.dropna()
In [129]:
dfff = dfff[['gross', 'duration', 'aspect_ratio', 'num_critic_for_reviews', 'director_facebook_likes', 'actor_3_facebook_likes', 'actor_1_facebook_likes', 'cast_total_facebook_likes', 'num_user_for_reviews', 'budget', 'actor_2_facebook_likes', 'imdb_score']];
In [130]:
dffff = dfff[['gross', 'budget','num_user_for_reviews','imdb_score']];
In [131]:
k_means = KMeans(init='k-means++', n_clusters=4, random_state=0)
In [132]:
k_means.fit(dfff)
Out[132]:
In [133]:
from yellowbrick.cluster import SilhouetteVisualizer
from yellowbrick.datasets import load_nfl
visualizer = SilhouetteVisualizer(k_means, colors='yellowbrick')
visualizer.fit(dfff) # Fit the data to the visualizer
visualizer.show()
Out[133]:
In [134]:
from scipy.spatial.distance import cdist
K = range(1, 10)
meandistortions = []
for k in K:
kmeans = KMeans(n_clusters=k, random_state=1)
kmeans.fit(dfff)
meandistortions.append(sum(np.min(cdist(dfff, kmeans.cluster_centers_, 'euclidean'), axis=1)) / dfff.shape[0])
plt.plot(K, meandistortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Average distortion')
plt.title('Selecting k with the Elbow Method')
Out[134]:
In [135]:
range_n_clusters = list (range(2,10))
print ("Number of clusters from 2 to 9: \n", range_n_clusters)
So from the graph we can see K=4 is the best and actually I had a good guess in first place. However we can consider K=5 as well but it won't improve much!¶
Story telling¶
This data sets gives us a very usefull information about IMDB scores of 5000 movies and their other related factors on the IMDB scors. These information gives us this opportunity to see what factor potentially gives high score to a movie and what factor is not important. The data set contained the following columns. To have better judgment about value of a movie in terms of money, I calculated profit (gross income - budget) later and added to the current columns¶
First of all to have a reliable analysis we need a reliable data set and to make sure we have a reliable data set we need to prepare our data. By inspecting data we noticed that we have some missing (as NaN or zero values) and duplicated values. after cleaning our data set shrinked to about 3700 movies which is still a big data set to do analysis. Also almost all of movies are color and just 123 of them are black and white. we can remove color column here but personally I don't like miss any information so I'll keep it.¶
Here is some analysis:¶
- This data set contain movies from several contries. USA, France, Germany, Canada, Australia, Spain, Japan, Hong Kong, China, Italy, New Zealand, South Korea, Denmark, Ireland, Mexico, Brazil, India, Iran and Norway are first 20 countries in this list.
- Interesting point about countries is India with that big Bollywood has just 5 movies in this list and countries Iran has 4 movies. Just a general information I know Bollywood makes more movies than Hollywood in a year!
- My client were more interested about movies from Iran so I digged into it and I found the names. There was 4 movies from Iran but 3 movuies in persian and that was because one old movie made in Iran and Afghanestan and mostly in English.
- Checking profit and IMDB score showed that a movie (Eddie the Eagle) had negative profit with IMDB score of 15 which was really high!!!
- Avatar had the best profit but The Dark Night had the best IMDB score!
- A huge number of movies has imdb better than 8 which I named as incredible
- I didn't see any strong relations between actors facebook likes and imdb score
- Correlation analysis showed that num_critic_for_reviews, duration, num_voted_users, num_user_for_reviews, and movie_facebook_likes are highly positive correlated to imdb score and title year is negativly and relativly correlated which means people probably won't like the most recent movies as much as they liked old movies but probably fact is each movies takes times to get more viewers and increase imdb score and older movies had enough time for that
- that num_critic_for_reviews, duration, num_voted_users, num_user_for_reviews, and movie_facebook_likes are highly positive correlated to imdb score and title year is negativly and relativly correlated which means people probably won't like the most recent movies as much as they liked old movies but probably fact is each movies takes times to get more viewers and increase imdb score and older movies had enough time for that.
- all the above regression models has same r-squared valed (.304) but the full model has the lowest mean squared value which means full model is better. Also all p-values are small so all the selected fatcors are significant.
- Gradient boosting had the best performance among 5 different classifier that I used with accruacy close to 70 percent and then decision tree had 62 percent accuracy. SVM gave 67 percent but actually it didn't work well based on confusion matrix. Both SVM and neural network had lazy performance which means they classified all test data set in one category (SVM classified as good and neural network classified as bad).
- Finally we can say to use Kmeans clustering algourithm, the number of clusters is 4.
More analysis has been provided in the report.¶
In [136]:
Image(url= "https://s.hdnux.com/photos/01/03/74/10/17809955/3/480x480.png")
Out[136]:
